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Abstract 

J  This  paper  presents  an  algorithm  for  analyzing  a  standard  color  image  to  determine  intrinsic  images 
of  the  amount  of  interface  (^specular and  body  (^ittuse^f reflection  at  each  pixel.  The  interface 
reflection  represents  the  highlights  from  the  original  image,  and  the  body  reflection  represents  the 
original  image  with  highlights  removed.  Such  intrinsic  images  are  of  interest  because  the  geometric 
properties  of  each  type  of  reflection  are  simpler  than  the  geometric  properties  of  intensity  in  a  black- 
and-white  image. 

• 

The  algorithm  is  based  upon  a  physical  model  of  reflection  which  states  that  two  distinct  types  of 
reflection  -  interlace  and  body  reflection  --  occur,  and  that  each  type  can  be  decomposed  into  a 
relative  spectral  distribution  and  a  geometric  scale  factor.  This  model  is  far  more  general  than  typical 
models  used  in  computer  vision  and  computer  graphics,  and  includes  most  such  models  as  special 
cases.  In  addition,  the  model  does  not  assume  a  point  light  source  or  uniform  illumination 
distribution  over  the  scene. 

The  properties  of  spectral  projection  into  color  space  are  used  to  derive  a  new  model  of  pixel-value 
color  distribution,  and  this  model  is  exploited  in  an  algorithm  to  derive  the  intrinsic  images. 
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1.  Introduction 


When  we  look  around  us,  the  surfaces  we  see  are  typically  glossy.  They  may  seem  to  be  very  shiny, 
fairly  matte,  or  anywhere  in  between,  but  virtually  all  the  surfaces  around  us  exhibit  highlights  to 
varying  degrees.  These  highlights  are  most  pronounced  when  the  surface  normal  bisects  the  angle 
between  the  direction  of  illumination  and  the  direction  of  view,  making  the  position  and  intensity  of 
highlights  very  sensitive  to  viewing  geometry.  This  causes  problems  with  many  low-level  computer 
vision  methods  such  as  segmentation  (which  typically  assumes  uniform  or  smoothly  varying  intensity 
across  a  surface)  or  stereo  and  motion  analysis  (which  attempt  to  match  images  taken  from  different 
viewpoints). 

Highlights  are  not  the  only  source  of  intensity  variation  across  a  surface.  Even  with  uniform 
illumination  of  a  matte,  uniformly  colored  surface,  there  will  be  smooth  shading  due  to  the  angle  of 
incidence  of  the  incoming  illumination  relative  to  the  surface  normal. 

It  would  be  very  useful  to  be  able  to  separate  the  effects  of  shading  from  highlights.  This  might 
result  in  intrinsic  images  telling,  at  each  pixel,  the  intensity  of  the  shading  and  the  intensity  of  the 
highlight  at  that  point.  This  separation  was  first  suggested  by  Barrow  and  Tenenbaum  [1],  and  would 
effectively  produce  an  image  of  the  highlights,  and  an  image  with  the  highlights  removed.  These 
would  be  useful  for  analysis  since  each  of  these  phenomena  is  more  simply  related  to  the  angles  of 
illumination  and  viewing  than  is  their  sum  (which  is  measured  in  a  black-and-white  image).  In 
addition,  the  relative  insensitivity  of  shading  to  viewpoint  would  make  the  shading  intrinsic  image  an 
ideal  candidate  for  stereo  or  motion  analysis. 

It  is  frequently  observed  that  highlights  have  a  different  color  from  the  characteristic  color  of  a 
surface  (which  is  related  to  shading).  In  this  paper,  we  show  how  a  simple,  rather  general  model  of 
reflection,  called  the  "Dichromatic  Reflection  Model",  can  be  .used  to  determine  intrinsic  images  of 
the  two  types  of  reflection  from  a  standard  color  image  (i.e.  red-green-blue  separation). 

The  analysis  uses  the  properties  of  spectral  projection,  the  process  whereby  color  pixel  values  are 
determined  from  the  spectral  power  distribution  (SPD)  of  incoming  light.  Combining  the  Dichromatic 
Reflection  Model  with  spectral  projection  results  in  a  new  model  of  pixel  value  distribution  in  R-G-B 
color  space.  The  model  predicts  that  pixel  values  from  pixels  on  a  single  surface  will  lie  on  a 
parallelogram  in  color  space,  and  that  the  position  of  any  pixel’s  color  within  that  parallelogram  yields 
the  coefficients  of  the  two  types  of  reflection. 


A  simple  algorithm  is  presented  which  utilizes  this  model  of  pixel  values  to  determine  the  desired 
intrinsic  images.  The  model  and  the  algorithm  are  then  extended  to  deal  with  diffuse  illumination  and 
shadows.  Additional  work  for  the  future  includes  implementation  of  the  algorithm  and  verification  of 
the  model  with  real  images. 

For  additional  background  information,  the  reader  interested  in  radiometry  in  general  is  referred  to 
[9],  [12],  and  [22];  while  appearance  measurement  (gloss  and  color)  is  discussed  in  [10],  [14],  [16], 
and  [35]. 

1.1  Previous  Work  in  Color  Image  Understanding 

Color  image  understanding  in  the  past  has  not  been  based  on  general  models  of  reflection.  Most  of 
the  work  has  been  the  application  to  three-dimensional  color  space  of  algorithms  originally  developed 
and  used  for  analyzing  monochrome  images.  This  includes  primarily  edge  detection  [21]  and 
clustering  [5, 18, 20,  23]  (etc.).  In  such  work,  image  regions  or  edges  are  identified  by  distances 
between  pixel  values  in  color  space,  without  appeal  to  any  model  of  color  generation  in  the  scene. 

Color  has  also  been  used  for  object  labelling  based  on  known  object  colors  or  object  colors 
measured  in  the  image  [19,  26, 31,  32, 34,  36].  This  approach,  known  as  "spectral  signature  analysis” 
in  remote  sensing,  uses  the  known  reflection  properties  of  materials  of  interest  in  a  particular  domain. 
It  depends  on  having  few  different  types  of  materials  in  the  scene,  and  having  prior  knowledge  of  their 
spectral  reflectance. 

The  properties  of  color  space  transformations  have  also  been  studied  [17, 24],  although  no  such 
transformations  have  sufficed  to  "solve”  the  image  segmentation  or  labelling  problems. 

The  only  previous  work  in  utilizing  general  properties  of  reflected  color  has  been  in  the  form  of 
simple  statements  such  as  "far-away  outdoor  objects  look  bluish”,  "outdoor  shadows  are  bluish”, 
and  "natural  colors  tend  to  be  desaturated”  [19, 31].  In  the  psychological  field,  Rubin  and  Richards 
proposed  a  method  for  using  color  to  determine  changes  in  material  across  an  image  [27];  however, 
while  the  method  is  quite  interesting,  their  assumptions  appear  to  be  restrictive. 

The  approach  described  in  this  paper  is  significant  because  it  is  applicable  to  many  common  types 
of  materials,  without  prior  knowledge  about  their  colors,  and  because  it  is  based  on  a  very  general 
model  of  reflection  in  the  real  world. 

Section  2.3  presents  a  discussion  of  previous  work  in  reflectance  models  for  computer  graphics  and 
image  understanding. 


2.  Reflection  and  the  Dichromatic  Model 


In  this  chapter,  we  present  a  brief  account  of  the  physics  of  reflection,  followed  by  the  Dichromatic 
Reflection  Model  which  captures  certain  aspects  of  the  reflection  process.  The  validity  and 
assumptions  of  the  model  are  discussed,  and  it  is  compared  with  other  reflection  models  used  in 
computer  graphics  and  image  understanding. 


2.1  Physical  Properties  of  Reflection 

The  methods  of  this  paper  deal  with  materials  which  are  optically  inhomogeneous,  meaning  that 
light  interacts  both  with  a  medium  that  comprises  the  bulk  of  the  surface  matter,  and  with  particles  of 
a  colorant  that  produce  scattering  and  coloration.  Many  common  materials  can  be  described  this 
way,  including  most  paints,  varnishes,  paper,  ceramics  (including  porcelain),  plastics,  etc.  Materials 
that  are  homogeneous  are  not  included  in  this  discussion;  thus  metals  and  many  crystals  are  not 
amenable  to  the  analysis  presented  herein.  We  will  also  limit  this  discussion  to  opaque  surfaces, 
which  transmit  no  light  from  one  side  to  the  other. 
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Figu re  2- 1 :  Reflection  of  Light  from  an  Inhomogeneous  Material 


Although  we  sometimes  think  of  a  visual  surface  as  a  plane,  this  is  an  approximation  useful  only  at 
the  macroscopic  level.  To  understand  reflection  from  inhomogeneous  materials,  we  will  instead  view 
the  surface  as  having  a  definite  thickness  (Figure  2-1)  [14, 16, 35]. 


When  light  strikes  a  surface,  it  must  first  pass  through  the  interface  between  the  air  and  the  surface 


medium.  Because  the  medium's  index  of  refraction  differs  from  that  of  the  air,  some  of  the  light  will  be 
reflected  at  the  interface  producing  interface  reflection  as  shown  in  figure  2-1.  The  direction  of  such 
reflection  will  be  in  the  "perfect  specular  direction"  relative  to  the  local  surface  normal,  i.e.  reflected 
about  the  surface  normal.  Note  that  most  materials  are  optically  "rough",  with  local  surface  normals 
that  differ  from  the  macroscopic  or  reference  surface  normal.  The  local  perfect  specular  direction  is 
therefore  somewhat  different  from  the  macroscopic  perfect  specular  direction,  so  that  the  interface 
reflection  will  be  somewhat  scattered  at  the  macroscopic  level. 

The  amount  of  light  reflected  at  the  interface  is  governed  by  Fresnel's  laws,  which  relate  interface 
reflectance  to  the  angle  of  incidence  (relative  to  the  local  surface  normal),  the  index  of  refraction  of 
the  material,  and  the  polarization  of  the  incoming  illumination  [15].  Interface  reflectance  is  thus  a 
function  of  wavelength  of  light,  since  index  of  refraction  generally  depends  on  wavelength,  but  the 
amount  is  typically  constant  to  within  a  few  percent  across  the  visible  spectrum;  acrylic  plastic,  for 
example,  has  an  index  of  refraction  of  1.485  at  400  nm  (the  blue  end  of  the  visible  spectrum)  and 
1.505  at  700  nm  (the  red  end)  [3, 16,  35],  producing  Fresnel  reflection  coefficients  of  3.8%  and  4.1%  at 
the  ends  of  the  spectrum.  Interface  reflection  is  frequently  assumed  to  be  constant  with  respect  to 
wavelength  because  of  the  small  magnitude  difference,  and  is  thus  said  to  have  the  same  color 
(relative  SPD)  as  the  illuminant  [7, 14, 16].  The  effect  of  polarization  is  more  severe,  and  interface 
reflection  tends  to  be  highly  polarized,  especially  for  large  angles  of  incidence  ("grazing  angles") 
[IS,  35], 

The  light  that  penetrates  through  the  interface  passes  through  the  medium,  where  it  undergoes 
scattering  from  the  colorant,  and  eventually  is  either  transmitted  through  the  material  (if  it  is  not 
opaque),  absorbed  jy  the  colorant,  or  re-emitted  through  the  same  interface  by  which  it  entered, 
producing  body  reflection  as  shown  in  figure  2-1.  The  geometric  distribution  of  this  body  reflection  is 
sometimes  assumed  to  be  isotropic,  i.’e.  independent  of  viewing  direction,  although  some  work  is 
under  way  to  produce  improved  models  of  scattering  [7, 10],  The  color  of  the  body  reflection  is 
generally  different  from  that  of  the  illumination,  since  interactions  with  colorant  particles  result  in 
absorption  with  a  probability  dependent  on  wavelength.  Body  reflection  is  usually  considered  to  be 
unpolarized. 

Real  reflection  tends  to  be  more  complex  than  described  above.  For  example,  real  body  reflection 
is  not  isotropic  (figure  2-2  shows  curves  of  reflection  as  a  function  of  viewing  direction)  [13],  and 
some  materials  exhibit  several  interfaces  producing  interface  reflection  (Figure  2-3)  [30].  However, 
the  explanation  provided  above  is  a  very  useful  approximation. 
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Figure*  2-2:  Body  Reflection  is  Not  Really  Isotropic  (from  [13]) 

This  figure  chows  some  curves  of  reflection  as  a  function  of  viewing  angle  for  various  materials.  In 
each  case,  the  direction  of  illumination  is  held  constant  as  the  distribution  of  reflection  is  measured. 
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Figure  2-3:  A  Paint  Layer  Producing  Two  Interface  Reflections  (from  [30]) 
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2.1.1  An  Aside  on  Terminology 

This  paper  adopts  the  terms  "interface"  and  "body”  reflection  rather  than  the  more  common  terms 

"specular"  and  "diffuse"  reflection.  While  the  latter  have  gained  some  popularity  in  the  literature  of 

appearance  measurement  and  computer  imaging  (graphics  and  vision),  they  have  some  severe 

• 

problems  which  render  them  controversial.  "Specular  reflection"  may  mean  any  of  three  things: 

•  interface  reflection  -  the  most  common  usage 

•  interface  reflection  in  perfect  specular  direction  at  macroscopic  level  --  the  usage  most 
common  in  gloss  measurement 

•  interface  reflection  where  surface  is  locally  optically  smooth  the  usage  in  scattering 
theory  and  laser  optics  [2]. 

Similarly,  the  term  "diffuse  reflection”  means  "reflected  light  scattered  over  a  large  solid  angle",  and 
thus  may  apply  equally  well  to  body  and  to  interface  reflection.  The  author,  having  experienced 
numerous  communication  problems  due  to  the  use  of  the  terms  specular  and  diffuse,  has  ultimately 
adopted  the  terminology  used  in  this  paper,  which  is  sometimes  found  in  the  more  technical  optics 
literature. 


The  terminology  used  throughout  this  paper  for  describing  reflectance  geometry  is  illustrated  in 
figure  2-4,  which  defines  the  following  angles  [1 1]: 

•  the  angle  of  incidence,  i  -  the  angle  between  the  illumination  direction  f  and  the  surface 
normal  N 

•  the  angle  ofemittance,  e  ••  the  angle  between  Wand  the  viewing  direction  V 


•  the  phase  angle,  g  ••  the  angle  between  f  and  V 


•  the  off  specular  angle,  s  the  angle  between  V  and  the  direction  of  (macroscopic) 
perfect  specular  reflection  J. 

We  also  use  the  standard  symbol  X  to  refer  to  the  wavelength  of  light. 

2.2  The  Dichromatic  Reflection  Model 

We  now  propose  a  simple  mathematical  model  of  reflectance,  based  on  the  above  discussion, 
called  the  Dichromatic  Reflection  Model. 

The  Dichromatic  Reflection  Model  states: 

L  (X,  /,  e,  g)  *  Lj  (X,  i,  e,  g)  +  Lb  (X,  /,  e,  g)  (2-1) 

=  mi  (/,  e,  g)  cf  (X)  +  mb  (»,  e,  g)  cb  (X)  (2-2) 

This  model  represents  two  statements  about  reflected  light,  as  expressed  by  the  two  parts  of  the 
equation: 

1.  Equation  (2-1)  says  that  the  total  radiance  L  of  the  reflected  light  is  the  sum  of  two 
independent  parts:  the  radiance  Lt  of  the  light  reflected  at  the  interface  and  the  radiance 
Lb  of  the  light  reflected  from  the  surface  body. 

2.  Equation  (2-2)  says  that  each  of  these  components  of  the  light  can  be  decomposed  into 
two  parts: 

a.  composition  --  a  relative  spectral  power  distribution  c,  or  cb  which  depends  only  on 
wavelength  but  is  independent  of  geometry,  and 

b.  magnitude  -  a  geometric  scale  factor  mx  or  mb  which  depends  only  on  geometry 
and  is  independent  of  wavelength. 

Intuitively,  the  Dichromatic  Reflectance  Model  says  that  there  are  two  independent  reflection 
processes,  and  that  each  has  a  characteristic  color  whose  magnitude,  but  not  spectral  distribution, 
varies  with  the  directions  of  illumination  and  view. 

In  the  remainder  of  this  chapter  we  will  address  the  scope  of  this  model  and  its  validity;  in  the  next 
chapter  we  will  see  how  this  model  may  be  exploited  to  determine  intrinsic  images  of  m-t  and  mb,  the 
amount  of  interface  and  body  reflection  at  each  pixel. 

The  Dichromatic  Reflection  Model  assumes  the  following: 

•  The  surface  is  an  opaque,  inhomogeneous  medium  with  one  significant  interface. 

•  The  surface  is  not  optically  active,  i.e.  it  has  no  fluorescence  or  thin-film  properties,  and  it 
is  uniformly  colored,  i.e.  it  has  a  uniform  distribution  of  the  colorant. 

•  Reflection  from  the  surface  is  isotropic  with  respect  to  rotation  about  the  surface  normal. 
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•  There  is  no  inter-reflection  among  surfaces. 

•  There  is  a  single  light  source,  i.e.  no  diffuse  ("ambient")  illumination,  and  the  relative 
spectral  power  distribution  S(A)  of  the  illumination  is  constant  across  the  scene. 

The  assumptions  about  the  surface  are  typical  for  reflectance  models  and  not  too  unrealistic.  The 

assumption  of  no  inter- reflection  is  also  typical,  but  unfortunately  not  realistic  at  all.  Finally,  the 

assumption  of  no  ambient  light  is  not  at  all  realistic,  but  will  be  relaxed  (in  fact,  eliminated)  in  section 

4.1  below. 


Equally  interesting  is  a  list  of  assumptions  not  made  by  the  model,  which  express  the  scope  or 
generality  of  the  model: 

•  Imaging  geometry  -  the  model  makes  no  assumption  that  orthography  or  perspective  is 
being  used.  Either  projection  satisfies  the  model. 

•  Planar  surface  -•  the  model  applies  equally  to  curved  and  planar  surfaces.  It  also  applies 
to  textured  surfaces,  i.e.  surfaces  with  macroscopic  roughness  (but  see  the  note  below 
about  analyzing  intrinsic  images). 

•  Specific  reflectance  model  -  the  model  does  not  assume  specific  functions  nr,  cjt  mb,  or 
cb;  in  particular,  there  is  no  specific  geometric  model  of  highlights,  no  assumption  that 
the  highlights  have  the  same  color  as  the  illumination,  and  no  assumption  that  the  body 
reflection  is  isotropic. 

•  Point  light  source  -  the  model  applies  equally  well  to  a  point  light  source,  an  extended 
light  source,  or  a  light  source  infinitely  far  away. 

•  Uniform  distribution  of  illumination  -■  the  model  does  not  assume  that  the  amount  of 
illumination  is  the  same  everywhere  in  the  scene;  only  that  the  SPD  is  the  same  (see 
above).  This  is  important,  since  real  (especially  extended)  light  sources  produce 
nonuniform  amounts  of  illumination  in  different  areas  of  the  scene. 

It  must  be  pointed  out  that  any  complexity  of  the  forms  mentioned  above  will  cause  great  difficulty  in 

analyzing  the  resulting  intrinsic  images  of  m-t  and  mb.  However,  the  Dichromatic  Reflection  Model  will 

make  it  possible  to  compute  these  intrinsic  images  regardless  of  such  complexity. 


In  spite  of  the  apparent  simplicity  of  the  Dichromatic  Reflectance  Model,  it  suffers  from  some  flaws. 

In  practice,  they  ought  to  have  only  a  minimal  impact  on  the  usefulness  of  the  model. 

•  There  is  no  obvious  way  to  decide  how  to  scale  the  magnitude  functions  mt  and  mb 
against  the  composition  functions  c(  and  cb.  Therefore,  the  resulting  intrinsic  images  may 
be  only  of  relative,  rather  than  absolute,  reflection  magnitudes.  At  least,  it  is  reasonable 
to  require  that  0  £  mv  mb  £  1  for  all  /,  e,  and  g. 


•  Interface  reflection  exhibits  an  interdependence  between  wavelength  and  geometry,  as 
expressed  by  Fresnel’s  equations.  As  stated  above,  this  is  a  small  effect;  the  author’s 


estimate  is  an  effect  no  greater  than  2%  in  pixel  value  errors  if  the  Dichromatic  Model  is 
assumed.  An  error  so  large  would  only  occur  when  i  >  60°  or  so  and  the  viewing 
direction  is  nearly  the  perfect  specular  direction;  in  most  situations,  therefore,  the  effect 
should  be  negligible. 

•  Body  reflection  also  exhibits  an  interdependence  between  wavelength  and  geometry.  If 
Cj(A)  is  not  constant,  then  the  color  of  the  light  passing  through  the  interface  will  differ 
somewhat  from  the  color  of  the  illumination  S( A).  Since  the  total  amount  of  light  reflected 
at  the  interface  ( Je  fg  Lj[A,/',e,g]  dg  de)  varies  with  the  angle  of  incidence  /,  the  color  of 
the  light  passing  through  the  interface  into  the  material  body  will  also  vary  with  the  angle 
of  incidence.  Thus,  the  color  of  the  body  reflection  should  vary  with  geometry.  However, 
since  c;  is  generally  nearly  constant,  this  should  also  be  a  negligible  effect  in  real  images. 

In  summary,  the  Dichromatic  Reflection  Model  is  expressed  in  a  simple  equation  which  should  be 
approximately  true  over  a  very  wide  range  of  circumstances. 

2.3  Previously  Used  Reflection  Models 

Previous  models  of  reflection  used  in  computer  graphics  and  computer  vision  have  proposed 
specific  reflectance  functions  to  predict  the  exact  amount  of  reflection. 

The  most  frequently  cited  model  is  that  of  Phong  [25, 8],  also  applied  in  computer  vision  by  Horn 

HU: 

n  +  1  . 

L  (/,  e,  g)  a  t — - —  cos  s  +  (1  -  t)cosi 

Here,  t  and  n  are  parameters  of  the  material,  with  t  representing  the  total  amount  of  light  reflected  at 
the  interface  and  n  representing  the  clustering  of  the  interface  reflection  about  the  perfect  specular 
direction.  The  equation  shown  above  is  only  a  monochrome  model,  but  is  frequently  applied  for  color 
images  with  the  color  of  the  interface  reflection  being  white  and  the  color  (R-G-B  values)  of  the  body 
reflection  being  a  parameter  of  the  material.  Phong's  model  is  easily  seen  to  be  an  instantiation  of  the 
Dichromatic  Reflection  Model  in  this  paper  as  follows: 
n  +  1 

/n(  *  t — - — cos  s  mb  ■  (1  -  t)cosi 

c,  •  1  cb  =  surface  parameter 

A  later  model  was  Blinn’s  [4],  in  which  the  interface  reflection  was  characterized  by  a  facet  model  of 
surface  structure  (from  [33])  and  Fresnel’s  equations  were  used  for  the  color  of  the  interface 
reflection.  Cook  and  Torrance,  in  more  recent  work  [6],  used  Beckmann’s  model  of  reflection  from  a 
rough  surface  [2]for  the  distribution  interface  reflection,  and  similarly  applied  Fresnel’s  equations  for 
the  color  of  the  interface  reflection.  In  both  cases,  the  body  reflection  was  assumed  to  be  isotropic, 


i.e.  mb  a  cos  i.  These  models  of  reflection  are  also  examples  of  the  Dichromatic  Reflection  Model, 
with  the  exception  of  the  use  of  Fresnel’s  equations  to  govern  the  color  of  the  interface  reflection 
(usually  a  small  effect,  as  noted  above). 

In  computer  vision,  Shibata  and  Frei  [29]  used  the  following  model  to  predict  reflection  in  aerial 
images: 

L  (»,  e,  g)  =  t1  cosn  s  +  l2  cosm  g  +  (1  -  f,  -  t2)  cos  i 
Here,  the  first  and  last  terms  are  similar  to  Phong's,  and  the  middle  term  represents  "backscattering" 
of  light  in  the  direction  of  illumination.  The  constants  t2,  n,  and  m  are  parameters  of  the  material 
being  viewed.  Extended  to  color  images,  this  model  would  also  be  an  instantiation  of  the  Dichromatic 
Reflection  Model,  with  the  backscattering  term  added  to  nij. 


3.  Pixel  Values  in  Color  Space 

This  chapter  begins  with  a  discussion  of  spectral  projection,  the  relationship  between  an  SPD  of 
light  and  its  color  coordinates.  When  this  process  is  applied  to  the  Dichromatic  Reflection  Model,  a 
new  model  for  pixel  values  in  color  space  is  the  result.  A  simple  algorithm  is  presented  for  exploiting 
this  model  to  determine  the  intrinsic  images  of  and  mb. 

3.1  Spectral  Projection 

Spectral  projection  is  the  process  whereby  pixel  values  are  computed  from  the  spectral  power 
distribution  (SPD)  of  the  measured  light.  The  process  in  a  monochrome  camera  is  quite  simple,  with 
the  pixel  value  p  being  just  a  summation  of  the  amount  of  light  at  each  wavelength  X(A),  weighted  by 
the  responsivity  of  the  camera  to  the  various  wavelengths,  s(X): 

p  =  /  X(\)  s(A)  dX 

The  interval  of  summation  is  determined  by  s(A),  which  is  non-zero  over  a  bounded  interval  of 
wavelengths  X. 

In  a  color  camera,  color  filters  are  interposed  between  the  incoming  illumination  and  the  camera. 
Each  filter  has  a  transmittance  function  r(A),  specifying  the  fraction  of  light  transmitted  at  each 
wavelength;  thus,  spectral  projection  with  a  filter  is  specified  by  the  above  integral,  with  s(X)  retplaced 
by  t(A)  s(A).  Typically,  three  filters  (red,  green,  and  blue)  are  used  with  transmittances  rr,  rB.  and  rb, 
resulting  in  a  vector  of  three  color  values,  C  ■  [r,  g,  bj.  If  we  let  r(X)  be  the  responsivity  of  the 
camera  combined  with  the  red  filter,  r(A)  =  rr(A)  s(A),  (etc.  for  g  and  b),  then  the  color  value  of  any 
SPD  X(A)  is  given  by: 


fx 

/*( X)  r(A)dX  ' 

9x 

iX(X)  g(X)  dX 

L6x  J 

LJx(X)  b(X)  dX 

Spectral  projection  is  a  linear  transformation,  as  shown  in  [28];  in  other  words,  CaX+bY  «  aC*  + 
bCy  where  a  and  b  are  scalars  and  X(A)  and  Y(X)  are  SPD's.  To  see  this,  consider  first  the  red 
component  r^ + bY;  it  is  easily  seen  that: 

rax+bY  “  /  laXW  +  byWl  r(A)  dX  =  a  /  X(\)  r(X)  dX  +  b  /  Y(X)  r(X)  dX 
*  a  rx  +  b  ry 


With  similar  equations  for  green  and  blue,  we  have  the  complete  result.  The  linearity  property  of 


12 


spectral  projection  is  important  because  it  says  that  a  mixture  of  two  SPD's  of  light  results  in  a  sum  of 
the  corresponding  pixel  values,  taken  in  the  same  proportion. 

3.2  The  Dichromatic  Model  in  Color  Space 

When  the  linear  property  of  spectral  projection  is  combined  with  the  Dichromatic  Reflection  Model, 
a  powerful  new  model  of  pixel  color  values  results. 

First,  consider  a  specific  point  on  a  surface.  At  that  point,  the  geometry  (angles  /,  e,  and  g)  is 
determined,  and  the  magnitudes  mi  and  mb  may  be  considered  as  scalars.  So,  the  Dichromatic 

Reflectance  Model  may  be  rewritten  at  a  specific  point  as: 

L  (A)  -  ♦  mbcb(A) 

This  defines  the  SPO  of  the  light  reflected  from  the  surface.  Now,  applying  the  linearity  of  spectral 

protection,  we  have: 

q  -  rn(  q  ♦  rnb  q, 

where  e  the  color  (pixel  value)  measured,  mi  and  mb  are  the  magnitudes  of  reflection  at  the  point 
m  question,  and  q  and  q,  are  the  colors  of  the  interface  and  body  reflection  of  the  material. 

Consider  the  colors  of  a  set  of  points  on  the  same  uniformly  colored  surface.  Because  the 
geometry  is  different  at  each  point,  the  scale  factors  mt  and  mb  vary  from  point  to  point.  However,  the 
colore  q  and  of  the  interface  and  body  reflection  are  the  same  at  all  points  on  the  same  surface, 
because  they  are  simply  the  spectral  projections  of  Cj(A)  and  cb(A)  which  do  not  vary  with  geometry. 
In  other  words,  the  pixel  values  are  a  linear  combination  of  q  and  q,,  with  the  coefficients 
determined  by  mt  and  mb  at  each  point. 


Figure  3*  1 :  Pixel  Values  on  a  Surface  Lie  on  a  Parallelogram  in  Color  Space 
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Figure  3*2:  Position  Within  Parallogram  Determines  Magnitudes 


Recalling  that  we  can  assume  0  <  mit  mb  s  1  without  loss  of  generality,  we  see  that  the  pixel  values 
for  a  set  of  points  on  a  single  surface  must  lie  within  a  parallelogram  in  color  space,  bounded  by 
the  colors  Ct  and  Cb  of  the  interface  and  body  reflection  of  the  surface  (Figure  3-1).  One  corner  of 
this  parallelogram  will  be  located  at  the  origin,  [r,g,b]  a  [0,0,0].  Further,  within  this  parallelogram,  the 
position  of  any  color  is  determined  by  the  values  of  m.  and  mb  at  the  corresponding  point  (Figure  3-2). 


The  Dichromatic  Model  in  Color  Space  makes  the  following  assumptions: 

•  Validity  of  the  Dichromatic  Reflection  Model,  i.e.  all  the  assumptions  made  therein. 

•  Prior  segmentation  of  the  image  into  groups  of  pixels  known  (or  believed)  to  lie  on  a 
single  surface. 

•  Pixel  values  returned  by  the  camera  are  linearly  related  to  the  irradiance  on  the  sensor 
plane,  i.e.  the  camera  is  photometrically  calibrated.  If  the  camera  is  calibrated  for 
monochrome  response,  it  is  not  necessary  to  re  calibrate  it  with  each  color  filter 
separately  since  the  relative  SPD  of  a  filter's  transmittance  is  constant  with  respect  to 
total  intensity  of  illumination. 


The  model  of  pixel  values  does  not  assume: 

•  Specific  C,  -  it  is  not  assumed  that  C,  is  achromatic  (i.e.  r,  ■  g8  *  b),  nor  that  it  is  the 
same  for  all  surfaces  in  the  image. 

•  Specific  color  responses  ••  no  assumption  is  made  about  the  color  filters  rr(X),  r  (A),  and 
rb(\),  except  that  they  are  linearly  independent,  nor  is  any  assumption  made  about  the 
camera's  spectral  responsivity  s(X),  except  that  the  relative  spectral  responsivity  is 
constant  with  respect  to  the  total  amount  of  irradiance  and  constant  across  the  sensor 
plane. 


3.3  Analyzing  Color  Pixel  Values 

We  can  now  present  a  simple  algorithm  for  computing  the  intrinsic  images  of  m ( and  mb  over  a  set  of 
pixels  corresponding  to  a  single  surface  in  the  image: 

1.  Histogram  the  pixel  values  in  color  space. 

2.  Fit  a  plane  in  color  space  to  these  points,  with  the  restriction  that  the  plane  must  pass 
through  the  origin. 

3.  Fit  a  parallelogram  on  this  plane  to  the  points,  with  the  restriction  that  one  vertex  must  lie 
at  the  origin.  The  sides  of  this  parallelogram  are  C,  and  Cb.  If  it  is  necessary  to 
distinguish  one  from  the  other,  either  geometric  criteria  may  be  used  or  the  fact  that  Cj 
probably  lies  close  to  the  achromatic  axis  of  the  color  space. 

4.  At  each  pixel,  express  its  color  as  a  linear  combination  of  Ct  and  Cj,.  This  can  be 
performed  with  a  simple  linear  transformation,  i.e.  a  matrix  multiplication  of  the  [r,  g,  b) 
color  values  by  a  2x3  matrix  (not  3x3  since  the  direction  orthogonal  to  the  plane  in  color 
space  is  of  no  interest  except  as  an  error  term). 

5.  The  resulting  values  are  the  magnitudes  mi  and  mb. 


3.3.1  Assumptions  of  the  Dichromatic  Analysis  Algorithm 
In  addition  to  the  assumptions  of  the  Dichromatic  Model  in  Color  Space,  this  algorithm  depends 
upon  a  number  of  additional  assumptions  for  its  effectiveness: 

•  Cj  and  Cb  must  be  linearly  independent,  i.e.  the  interface  and  body  reflection  must  have 
different  colors. 

•  The  noise  in  measuring  pixel  values  must  be  small  enough  that  plane-fitting  and 
parallelogram-fitting  can  proceed. 

«  The  distribution  of  pixel  values  within  the  parallelogram  must  not  be  pathological.  In 
general,  the  author  expects  that  many  points  will  lie  close  to  the  Cb  axis,  i.e.  have  m,  close 
to  zero,  and  the  remaining  points  will  lie  close  to  the  C|  axis.  These  latter  points  may  have 
pixel  values  "clipped"  by  the  extent  of  the  color  cube  itself,  since  highlights  may  be  very 
bright  and  thus  cause  truncation  of  the  measured  pixel  value  at  the  maximum  response  of 
the  camera. 


It  should  also  be  noted  that  the  Dichromatic  Analysis  Algorithm  uses  pixel  colors  to  determine  m, 
and  mb,  but  it  does  not  interpret  m ,  and  mb  in  terms  of  the  photometric  angles  »,  e,  and  g.  Thus,  this 
algorithm  does  not  tell  how  to  analyze  the  values  of  and  m„. 
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3.4  Summary 

In  this  chapter,  we  have  seen  a  mathematical  description  of  spectral  projection  -•  the  process  by 
which  an  SFD  of  light  is  converted  into  color  values.  Spectral  projection  is  a  linear  transformation,  a 
fact  that  can  be  applied  to  the  Dichromatic  Reflection  Model  to  produce  a  model  of  pixel  values  in 
color  space.  The  distribution  of  pixel  values  is  expected  to  form  a  parallelogram,  bounded  by  the 
colors  Cj  and  ^  of  the  interface  and  body  reflection  from  the  surface.  The  position  of  any  color 
within  this  parallelogram  can  be  used  to  determine  the  coefficients  rrij  and  mb  at  the  corresponding 
pixel  in  the  image. 


4.  Applying  the  Dichromatic  Analysis  Algorithm 

In  this  chapter  we  discuss  the  extension  of  the  Dichromatic  Reflection  Model  for  dealing  with  diffuse 
illumination,  and  some  possible  methods  of  analyzing  the  intrinsic  images  of  /n{  and  mb. 

4.1  Extending  the  Model  for  Diffuse  Illumination 

The  Dichromatic  Reflection  Model  as  presented  above  depends  on  the  assumption  that  the 
illumination  at  any  point  comes  from  a  single  (point  or  extended)  light  source.  It  is  more  realistic  to 
model  the  illumination  as  consisting  of  a  light  source,  plus  "ambient"  or  "diffuse”  light,  of  lower 
intensity,  coming  from  all  directions  in  equal  amounts,  and  possibly  with  a  different  SPD  than  the 
small  source.  This  model  is  a  better  approximation  of  daylight,  which  contains  light  from  a  point 
source  (the  yellowish  sun)  and  light  from  a  hemisphere  (the  bluish  sky),  and  of  light  in  a  room,  which 
comes  from  light  fixtures  and  from  inter- reflections  off  walls  and  other  objects.  All  the  previously 
mentioned  models  of  reflection  in  computer  graphics  presume  some  sort  of  diffuse  illumination, 
although  the  assumption  made  here  (diffuse  illumination  from  all  directions  equally)  is  admittedly 
highly  idealized. 


pixel  colors 


Figu re  4- 1 :  Color  Space  Parallelogram  With  Diffuse  Illumination 

The  light  reflected  by  diffuse  illumination  contains  a  part  due  to  interface  reflection  and  a  part  due  to 
body  reflection.  By  assuming  that  this  light  is  incident  from  and  reflected  into  all  directions  equally,  it 
can  be  modelled  by  adding  a  single  term,  La(X),  to  the  Dichromatic  Reflection  Model: 

L  (X,  /,  e,  g)  •  m,  (/,  e,  g)  Cj  (X)  +  mb  (/,  e,  g)  cb  (X)  +  La(X) 

In  color  space,  the  model  becomes: 


Cl  =  n)j  Cj  +  mb  Cjj  +  Ca 

where  Ca  is  the  color  of  the  light  reflected  from  diffuse  illumination  La(X).  Since  Z.a(A)  does  not  vary 
with  geometry,  the  effect  of  this  change  is  to  translate  the  parallelogram  of  pixel  colors  for  a  single 
surface  away  from  the  origin  by  the  vector  Ca,  as  seen  in  figure  4-1 . 

The  Dichromatic  Analysis  Algorithm  can  still  be  applied  with  this  change,  but  note  that  the  plane¬ 
fitting  and  parallelogram-fitting  operations  specified  in  the  algorithm  will  be  less  constrained  and 
therefore  more  error-prone,  if  the  relative  SPD  of  the  diffuse  illumination  is  the  same  as  that  of  the 
small  source,  then  Ca  is  a  linear  combination  of  Cj  and  Cb  [6];  in  such  a  case  the  plane  containing  the 
points  should  pass  through  the  color  space  origin  although  the  parallelogram  will  not. 

It  is  interesting  to  note  that  a  point  whose  color  lies  at  exactly  CL  =  Ca  has  mt  =  mb  =  0,  and  might 
therefore  be  suspected  of  lying  within  a  shadow.  This  is  a  far  more  precise  description  of  colors 
within  shadows  than  statements  of  the  form  "shadows  tend  to  be  bluish",  which  have  been  seen  in 
previous  work  in  image  understanding.  In  fact,  since  for  different  surfaces  Ca  is  very  likely  to  differ,  it 
may  be  possible  to  associate  shaded  parts  of  surfaces  with  illuminated  portions,  by  constructing  Ca 
for  each  adjacent  illuminated  area  and  finding  which  one  matches  best  with  the  color  of  the  pixels  in 
the  shaded  area.  Unfortunately,  since  pixel  values  in  shaded  areas  of  an  image  tend  to  be  poorly 
measured  by  current  digitizing  cameras,  this  kind  of  analysis  may  prove  to  be  unreliable. 

The  description  presented  here  is  intended  to  model  diffuse  illumination;  however,  it  will  be  a  poor 
substitute  for  a  detailed  model  of  inter- reflection  when  surfaces  are  close  to  each  other. 

The  author  continues  to  call  the  above  equation  a  "Dichromatic  Reflection  Model”  (rather  than  a 
trichromatic  model)  because  the  essence  of  the  model  is  that  reflection  occurs  from  two  places  in  a 
surface:  the  interface  and  the  surface  body. 

4.2  Analysis  of  the  Intrinsic  Reflection  Images 

The  potential  utility  of  the  intrinsic  images  of  m(  and  mb  stems  from  two  facts: 

•  Both  m,  and  mb  have  simpler  geometric  properties  than  does  intensity  in  a  monochrome 
image,  which  represents  a  weighted  sum  of  the  two. 

•  mb,  representing  diffusely  distributed  light,  is  relatively  insensitive  to  changes  in 
viewpoint  from  one  image  to  the  next. 

Here  are  some  possible  methods  for  exploiting  these  properties  of  the  intrinsic  images  of  m,  and  mb: 

1.  In  stereo  or  motion  analysis,  image  matching  using  mb  might  be  more  reliable  because  of 
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the  elimination  of  interface  reflection,  whose  position  in  the  image  is  highly  viewpoint- 
dependent. 

2.  If  a  specific  reflectance  model  such  as  Phong's  is  assumed,  then  the  two  values  rni  and 
mb  provide  two  constraints  on  the  surface  normal  at  any  point.  For  example,  using 
Phong’s  model,  the  angle  s  can  be  calculated  from  mt  and  the  angle  /  can  be  calculated 
from  mb.  With  such  constraints,  unique  surface  normals  can  be  established  from  a  color 
image.  (In  practice,  this  kind  of  analysis  would  be  limited  by  the  applicability  of  such 
reflectance  models  and  by  the  distribution  of  points  on  the  parallelogram  in  color  space.) 


3.  If,  as  above,  a  specific  reflectance  model  is  assumed  but  the  parameters  (t,  n  in  Phong’s 
model)  are  not  known,  it  may  be  possible  to  determine  or  estimate  these  from  the  image. 
At  a  point  with  a  known  orientation,  the  coefficients  ml  and  mb  can  be  determined  by 
color  analysis;  using  these,  the  surface  parameters  may  be  calculated.  This  might  form 
the  basis  for  a  technique  of  looking  at  parts  of  a  surface  (say,  the  spots  with  pronounced 
highlights)  to  estimate  the  nature  of  the  reflectance  properties  of  the  surface,  then  using 
this  estimated  reflectance  model  to  analyze  the  rest  of  the  surface  in  detail. 


4.  Using  differentials,  since  the  photometric  angles  /',  e,  and  g  are  functions  of  x  and  y  in  the 
image,  we  can  relate  the  differentials  of  the  intrinsic  image  of  mi  to  the  geometric 
properties  of  and  the  imaging  geometry  by: 


[dm/dx  dm/dy]  =  [dm/di  dm/de  dm/dg] 


di/dx  di/dy 
de/dx  de/d y 
dg/dx  dg/dy 


This  equation  tells  how  the  intrinsic  image  and  a  reflectance  model  yield  constraints  on 
the  imaging  geometry  using  first  derivatives. 
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5.  Conclusions 

In  this  paper,  we  have  seen  the  development  of  a  fairly  simple  algorithm  for  determining  intrinsic 
images  of  reflection  from  a  color  image.  The  presentation  included  a  fairly  general  reflection  model 
based  on  a  physical  description  of  the  reflection  process,  a  model  of  pixel  value  distribution  based  on 
the  properties  of  spectral  projection,  and  an  algorithm  for  analyzing  color  values  in  an  image.  This 
paper  atso  presented  a  discussion  of  some  related  issues  such  as  diffuse  illumination,  image 
segmentation,  and  analysis  of  the  resulting  intrinsic  images.  These  intrinsic  images  promise  to  be 
useful  for  a  variety  of  image  understanding  tasks,  including  stereo  matching,  geometric  surface 
analysis,  and  analysis  of  surface  reflectance  parameters. 

This  work  represents  a  quantum  advance  over  previous  efforts  in  color  image  understanding, 
because  it  presents  a  quantitative  theory  based  on  the  physics  of  the  real  world.  In  this  respect,  it 
follows  an  increasingly  important  paradigm  of  three-dimensional  image  understanding  research: 

1.  Describe  a  physical  (geometric  or  optical)  phenomenon  in  the  scene. 

2.  Develop  a  mathematical  statement  of  this  scene  property. 

3.  Apply  an  imaging  model  to  this  statement,  to  produce  a  new  mathematical  statement 
relating  image  properties  to  scene  properties. 

4.  Develop  an  algorithm  to  analyze  the  scene  properties  using  the  above  relationship. 

As  is  frequently  the  case,  the  complex  intermediate  mathematics  are  used  in  this  paper  only  to  derive 
the  algorithm;  they  are  not  necessarily  directly  incorporated  in  any  actual  program. 

This  paper  also  displays  one  of  the  dilemmas  of  current  work  in  3D  computer  vision:  we  frequently 
see  simple  theories  and  algorithms  developed  to  analyze  isolated  phenomena  such  as  shadows  and 
highlights,  but  we  rarely  see  them  applied  to  real  images.  While  some  readers  may  feel  that  this  is 
because  the  theories  are  too  complex,  the  opposite  is  a  more  tenable  point  of  view:  that  such 
theories  fail  in  practice  because  they  are  too  simple,  i.e.  their  assumptions  are  too  restrictive.  In  order 
to  apply  them  to  real  images,  it  is  first  necessary  to  relax  as  many  assumptions  as  possible,  as  was 
done  here  in  discussing  diffuse  illumination  and  image  segmentation.  This  produces  the  striking 
paradox  of  complex  theories  whose  validity  should  be  tested  by  applying  them  to  real  images;  but  they 
cannot  be  applied  to  real  images  until  they  are  made  yet  more  complex) 

We  have  yet  to  see  how  researchers  in  three-dimensional  image  understanding  will  resolve  this 
problem,  but  the  utility  of  such  research  depends  upon  its  solution.  The  current  work  may  help  by 
demonstrating  a  theory  which  depends  upon  relatively  few  special  assumptions,  and  by  providing 


"simpler”  intrinsic  images  useful  for  estimating  surface  and  illumination  parameters  from  examination 
of  the  images  themselves. 

Further  efforts  in  developing  this  work  will  include  implementation  (when  a  calibrated  camera  is 
obtained),  pursuing  the  ideas  for  intrinsic  image  analysis  suggested  above,  and  developing  the  theory 
that  will  ultimately  allow  analysts  of  nonuniform  illumination,  extended  light  sources,  and  perhaps 
inter- reflection  among  surfaces.  In  addition,  a  segmentation  scheme  based  on  the  Dichromatic 
Reflection  Model  might  be  investigated. 
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